3D Computational Ghost Imaging 
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Computational ghost imaging retrieves the spatial information of a scene us- 
ing a single-pixel detector. By projecting a series of known random patterns 
and measuring the back-reflected intensity for each one, it is possible to recon- 
struct a 2D image of the scene. In this work we overcome previous limitations 
of computational ghost imaging and capture the 3D spatial form of an object 
by using several single-pixel detectors in different locations. From each detec- 
tor we derive a 2D image of the object that appears to be illuminated from a 
different direction, using only a single digital projector as illumination. Com- 
paring the shading of the images allows the surface gradient and hence the 3D 
form of the object to be reconstructed. We compare our result to that obtained 
from a stereo-photogrammetric system utilizing multiple high-resolution cam- 
eras. Our lowcost approach is compatible with consumer applications and can 
readily be extended to non- visible wavebands. 
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Computational ghost imaging (GI) is an alternative technique to conventional imaging and 
removes the need for a spatially resolving detector. Instead, ghost imaging infers the scene by 
correlating the known spatial information of a changing incident light field with the reflected 
(or transmitted) intensity. 

The principles of GI were originally demonstrated using spatially entangled photon pairs 
produced by spontaneous parametric down-conversion, known as quantum GI (1, 2). The two 
photons propagate along different paths: in the first path, the photon interacts with the object 
and if not absorbed is detected by a detector with no spatial resolution, in the second path the 
photon never interacts with the object, but its transverse position is measured by a scanning 
imaging system. It is by correlating the coincidence measurements over many photon pairs that 
enables an image of the object to be deduced. 

It was subsequently demonstrated that GI could be performed not only using an entangled 
light source but also with thermal light, a technique commonly termed classical GI (3-6)). In 
classical GI a copy of the light field is usually made with a beam splitter, one copy of the 
light field interacts with the object and a non spatially resolving detector and the other copy is 
recorded with a camera. Correlations between the two detectors again yield an image, albeit 
one with a higher background than in the quantum case (7). The earlier controversy over the 
distinction between classical and quantum ghost imaging is now largely resolved (8). 

Classical GI systems can be simplified by introducing a device capable of generating com- 
puter programmable light fields, which negates the requirement for the beam splitter and the 
camera since knowledge of the light field is held in the computer memory. This type of sys- 
tem, termed computational GI (9, 10), has previously been performed using a programmable 
spatial light modulator (SLM) and a laser, but can also be achieved using a presentation-type 
programmable light projector. We note that in this form ghost imaging is related to the field of 
single-pixel cameras (11), the difference being an interchange of the light source and detector. 
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In both single pixel cameras and ghost imaging systems, inverting the known patterns and 
the measured intensities is a significant computational problem. A number of sophisticated 
algorithms have been developed over the years to improve the signal-to-noise ratio (SNR) for 
different GI systems (12, 13) but with appropriate normalization (14) a simple iterative algorithm 
was adopted for this experiment. 

All previous GI experiments have been restricted to relatively small 2-dimensional (2D) 
images, mainly of 2D template objects or 2D outlines of 3-dimensional (3D) objects. In this 
work we overcome previous limitations of computational GI and capture the 3D spatial form of 
an object by using several single-pixel detectors in different locations. A 2D image is derived 
from each detector but appears as if it is illuminated differently. Comparing the shading infor- 
mation in the images allows the surface gradient and hence the 3D form of the surface to be 
reconstructed. 

The experimental setup, illustrated in Fig]TJ consists of a digital light projector (DLP) (Texas 
Instruments DLP LightCommander) to illuminate objects with random binary light patterns, 
four spatially separated single -pixel photodetectors to measure the intensity of the reflected 
light, an analogue-to-digital converter to digitize the photodetector signals and a computer to 
generate the random speckle pattern as well as perform 3D reconstructions of the test object. 

The digital light projector comprises of a red, green and blue light emitting diode illumi- 
nation source and a digital micro-mirror device (DMD) to generate the structured illumination. 
The potentially large operational bandwidth of the DMD (300 nm-2 /mm) however would en- 
able the use of this technique at other wavelengths that are potentially unsuitable for existing 
imaging technologies. 

The projected patterns we use are randomly distributed binary patterns, having a black and 
white ratio 1 : 1, which are projected onto the object using a Nikon 50mm focal length lens. 
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Figure 1: Illustration of the experimental setup used for 3D surface reconstructions. The light 
projector illuminates the object (head) with computer generated random binary speckle patterns. 
The light reflected from the object is collected on 4 spatially separated single pixel photodetec- 
tors. The signals from the photodetectors are measured by the computer via the analogue-digital 
converter, and used to reconstruct a ghost image for each photodetector. 

The life-sized mannequin head is positioned about lm from the lens so that it fits within the 
projected pattern. Four spatially separated single-pixel photodetectors are positioned in the 
plane of the lens, separated by 500mm, to record the back-scattered light. For every binary 
pattern projected, the corresponding object intensity is measured by each photodetector, which 
is fed to a computer algorithm. 

DMD-based projectors create colour images by displaying 24 binary images (bit planes) per 
frame in quick succession. By alternating between a binary pattern and its inverse in subsequent 
bit planes we can demodulate the measured signal at the frequency of the bit plane projection 
(1440Hz) to isolate the back-reflected signal from light sources at other frequencies such as 
room lighting. Importantly, the fact that the speckle pattern has equal numbers of black and 
white pixels enables normalization of the measured signals for each pattern, which has been 
shown to improve the SNR of the final reconstruction (14). 
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In all iterative GI techniques a 2D representation of the object is reconstructed by averaging 
the product between the measured photodetector signal and the incident speckle pattern over 
many patterns. A sequence of N binary patterns, Pi(x, y) are reflected from the object, giving 
a sequence of measured signals S{. The 2D reconstruction, I(x, y), which provides an estimate 
of the object, can be found as 

I(x, y) = (($ - (Si)) (Piix, y) - (P^x, y)))) , (1) 

where < . >= denotes an ensemble average for iV iterations. 

Using Eq. [T]we obtain 2D reconstructions of the object for each of the four photodetectors 
employed in our system, shown in Fig. [2] The spatial information in each image is identical, 
however the apparent lighting of the scene is dependent on the location of the detector used 
to record the back- scattered light (optical imaging systems are reciprocal). Thus, in contrast to 
imaging systems based on multiple cameras, the perspective of the single pixel detector does not 
render geometrical distortion to the object being imaged. Instead the location of the detectors 
introduces the effective lighting position for the scene. 
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left right top bottom 

Figure 2: Source ghost images from each photodetector employed in the system, reconstructed 
using the TGI iterative algorithm iteratively from 1000 to 1,000,000. The spatial information in 
each image is identical, however the apparent illumination source is determined by the location 
of the relevant photodetector, indicated underneath. 
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Depth information of a scene is otherwise lost in a 2D image, but there are instances where 
it can be inferred using a technique called 'shape from shading' (SFS) (15, 16). From a single 
image, with one source of illumination, this method relies on the shading caused by geometrical 
features to reveal the depth of the scene. In general, most SFS methods assume the object ex- 
hibits uniform Lambertian (matte) reflectance and that a single light source is located at infinity, 
such that the incoming lighting vector is constant across the surface of the object. An extension 
of this technique, called 'photometric stereo' (PS) (77), adopts the same assumptions but uses 
multiple images, each with a different illumination, and taken from the same view point, similar 
to the types of images retrieved by our 3D computational GI system. 

The intensity of a pixel, I(x, y), in the image obtained from the n th detector can be expressed 

as 

I n (x,y) = a (L n -N), (2) 

where a represents the surface reflectance, L n is the unit illumination vector pointing from the 
detector to the object and N is the surface normal unit vector of the object. Thus for all images 
we can write Eq. [2] in matrix notation as 

I = a ■ L ■ N, (3) 
For any pixel (x, y) the unit surface normal is given 

N = (1/«)L" 1 I, (4) 

and the reflectance is given by 

a = |L -1 I|. (5) 

From the surface normals calculated at each pixel it is possible to determine the gradient be- 
tween adjacent pixels from which we obtain the surface geometry by integration. In fact, as we 
record four images, the problem becomes overconstrained as the surface normals represent only 
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two degrees of freedom per pixel. We can thus remove our assumption of uniform reflectivity 
and recover an estimate of the surface reflectance a at the same time as finding the object's 
shape. 

In our algorithm the surface gradients dz/dx and dz/dy were calculated from the surface 
normals for each pixel. Starting in the centre of the object and working outwards, the gradients 
were subsequently integrated to recover a depth-map of the object. The height for each pixel 
was estimated based on the height and the gradient at each of its nearest-neighbor pixels. Thus, 
the integration was performed iteratively, each pass over the object estimating the height of all 
the pixels having at least one nearest neighbor with a height estimate to work from. For pixels 
that have more than one nearest neighbor with an estimated height, the height is calculated 
from the average of those pixels. This simple algorithm works from the middle of the object 
outwards, and is capable of integrating around holes in the object where there is no information. 

Once the algorithm had been appropriately calibrated by imaging a flat surface, accounting 
for changes of the lighting vector for different pixels across the object plane, the system was 
tested for objects with geometric complexity. One object investigated was a life-size white 
polystyrene mannequin head, with approximate dimensions 190 x 160 x 250 mm. Using the 
2D images shown in Fig.|2]we calibrated for each pixel the reflectance, the surface normals, the 
surface gradient and the estimated depth as prescribed by our model. A standard 3D graphic 
package was then used to visualize this profile, overlaid with the reflectance data, as illustrated 
in Fig.[3j 
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Figure 3: Rendered views of the reconstructed facial surface derived by integration of the sur- 
face normal data and overlaid with the reflectance data (movie included in supplementary ma- 
terial). 

To quantify the accuracy of our approach the 3D reconstruction of the test object was com- 
pared with a 3D image captured from a stereo-photogrammetric camera system (Di3D®). This 
latter system uses a matching algorithm on the 2D images from multiple cameras to recover the 
distance map of an object from the cameras. The accuracy of this system with facial shapes is 
well documented (18) to have a root mean square (RMS) error of order 1mm. 

To compare the facial profiles measured by the two systems, the shapes are characterised 
by anatomical landmarks, which are well-defined facial locations (19) (for example, nose tip, 
mouth corners etc.). Figj4] shows two sets of 21 landmarks superimposed on the facial images 
by a trained observer. After lateral and angular registration and subsequent depth scaling, the 
RMS error of our ghost profiler is found to be slightly below 4mm. 
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(a) frontal view (b) profile view 



Figure 4: The matched ghost imaging (green) and stereo-photogrammetric (blue) reconstruc- 
tions of the mannequin head, from frontal (a) and profile (b) viewpoints, and with anatomical 
landmarks (colour coded green and blue respectively) added. 

Beyond showing that high-quality images of real life objects can be captured using a single- 
pixel photodetector, our experiment demonstrates that by using a small number of single-pixel 
detectors, computational ghost imaging methods can give 3D images. 

We have applied this 3D ghost imaging technique to record a facial shape, indicating good 
quantitative agreement with existing imaging technology based on stereo-photogrammetric sys- 
tems that employ several high-resolution cameras. An important difference in our approach is 
that a single projector determines the spatial resolution of the system, removing issues of pixel 
alignment associated with multiple cameras. Furthermore, reversing the fundamental imaging 
process allows for simpler, less expensive detectors to be utilised. The operational bandwidth 
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of the system is limited not by the efficiency of a pixelated imaging detector but instead by the 
reflectivity of DMD used for light projection, whose efficiency extends well beyond the visible 
spectrum. Development of such technology, for example the use of a broadband white light 
source, could enable GI systems to become a cheaper alternative for applications in 3D and 
multi- spectral imaging. 
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